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Abstract 

Seven  college  students  enrolled  in  a  college-level  introductory  oceanography  course  read 
and  answered  questions  on  two  selections  drawn  from  their  textbook  in  the  course.  Using  verbal 
protocol  procedures,  three  nonnative  English  speakers  and  four  native  English  speakers 
described  what  they  were  doing  to  answer  the  questions.  Students  also  varied  in  level  of 
expertise  based  on  their  backgrounds  in  related  science  courses.  The  questions  varied  in  terms 
of  their  relationship  to  the  text  and  the  type  of  processing  required  to  answer  them.  A  model  of 
question  answering  from  academic  texts  is  proposed  and  this  model  guided  protocol  analysis. 
Solution  strategies  were  abstracted  from  the  protocols  and  indicated  predicted  effects  of  question 
type  on  difficulty  and  on  solution  strategies.  Differences  between  individuals  were  related  to 
domain  expertise  and  to  language  background. 


This  report  is  an  expanded  version  of  the  paper  "Text  and  Learner  Characteristics  that  Affect 
Learning  from  Academic  Texts",  presented  at  the  meetings  of  the  Psychonomic  Society,  New 
Orleans,  Louisiana,  November,  1986.  This  research  was  sponsored  by  the  Personnel  and 
Training  Research  Programs,  Psychological  Sciences  Division,  Office  of  Naval  Research,  under 
Contract  No.  N00014-85-K-0562,  Contract  Authority  Identification  Number,  NR442c015. 
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It  has  become  almost  commonplace  to  assert  that  comprehension  is  an  interactive 
process  in  which  the  learner,  the  task  and  the  text  all  play  a  role  (e.g.  Brown,  Campione  &  Day, 
1981).  Over  the  past  year  and  a  half  we  have  been  conducting  research  that  attempts  to 
understand  and  detail  this  obvious  but  elusive  interaction  between  learner,  task  and  text 
characteristics.  The  purpose  of  the  present  report  is  to  describe  the  question-answering  task  we 
have  been  using  to  explore  these  interactions.  We  have  formulated  a  model  of  the 
question-answering  process  that  has  been  used  to  guide  the  coding  and  interpretation  of  verbal 
protocols  collected  from  students  who  vary  along  several  dimensions  known  to  have  important 
effects  on  learning.  Two  learner  characteristics  we  have  been  concerned  with  are:  language 
proficiency  of  nonnative  English  speakers  and  prior  knowledge.  We  are  focusing  on  language 
proficiency  as  it  relates  specifically  to  the  language  used  in  academic  texts.  What  proficiencies 
are  needed  when  attempting  to  learn  new  information  from  a  text?  What,  if  any,  problems  are 
unique  to  nonnative  English  speakers  confronted  with  this  task?  Prior  knowledge  is  of  interest  to 
us  particularly  as  it  interacts  with  language  proficiency.  High  versus  low  knowledge  effects  on 
comprehension  and  reasoning  have  been  demonstrated  by  a  number  of  researchers,  (e.g.  Chiesi, 
Spiiich,  &  Voss,  1979;  Dee  Lucas  &  Larkin,  1986;  Spilich,  Vesonder,  Chiesi,  &  Voss,  1979). 

Thus,  a  general  issue  with  which  we  are  concerned  is  the  interaction  of  language  proficiency  and 
domain-specific  knowledge. 

The  text  characteristics  upon  which  we  have  been  focusing  relate  to  local  and  global 
language  structures  that  are  used  to  convey  meaning  between  and  among  individual  units  of 
information.  Such  devices  impact  text  cohesion  and  the  learner's  ability  to  construct  a  coherent 
representation  of  the  text.  Examples  of  local  language  structures  are  conjunctions,  conditionals, 
performatives,  and  quantification  terms  and  phrases  (see  for  further  discussion  Celce-Murcia  & 


Larson- Freeman  1983).  Examples  of  global  text  structures  are  compare/contrast, 
thesis/evidence,  procedural  and  cause-effect.  These  global  structures  are  typically  signalled  by 
rhetorical  devices  at  the  paragraph  level  (e.  g.  Brewer,  1980;  Meyer,  Brandt,  &  Bluth,  1980). 


Finally,  we  have  been  looking  at  a  number  of  task  characteristics  that  arise  from  a 
consideration  of  comprehension  skill  hierarchies  (Rosenshine,  1980)  and  envisionment  levels 
(Fillmore,  1983;  Kay,  in  press;  Langer,  in  press).  Comprehension  skills  hierarchies  imply 
an  increasingly  more  sophisticated  understanding  as  one  moves  from  "literal"  comprehension  of 
the  text,  to  making  simple  inferences  from  the  text,  to  engaging  in  more  complex  inferential 
reasoning  based  on  the  information  in  the  text.  Comprehension  tasks  can  be  thought  of  as  varying 
along  a  continuum  reflecting  the  degree  to  which  the  task  can  be  successfully  completed  with 
"only"  the  text  as  compared  to  requiring  material  and  knowledge  external  to  the  text  (see  for 
discussion  Goldman,  1 985).  Variation  in  the  amount  of  text  dependency  is  also  reflected  in  the 
envisionment  levels  of  reasoning  proposed  by  Fillmore  (1983).  Envisionment  levels  refer  to 
variations  in  individuals'  understanding  of  the  world  described  in  a  text.  These  levels  range  from 
the  most  basic,  "Understanding  independent  statements  in  a  text",  to  the  most  complex, 
"Embellishing  the  text  world  in  light  of  existing  knowledge  and  in  terms  of  possible  extensions 
and  underlying  generalities",  'i  he  more  basic  levels  are  more  text  dependent,  literal 
understandings. 

With  respect  to  the  interaction  of  learner,  text  and  task  characteristics  we  have 
examined  the  comprehension  and  reasoning  performance  of  non-native  English  Speakers  (ESL) 
engaged  in  question  answering,  recall,  outlining  and  rewriting  tasks  with  oceanography  texts 
that  they  have  been  studying  in  connection  with  the  introductory  level  course  in  this  domain.  The 
four  tasks  demand  varying  levels  of  comprehension  and  reasoning  and  in  doing  the  tasks  students 
may  exhibit  different  degrees  of  dependency  on  the  text.  We  employed  a  think-aloud  protocol 
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methodology  to  examine  the  strategies  learners  employed,  given  their  levels  of  expertise  in  the 
domain,  their  English  language  proficiency  and  the  requirements  of  the  specific  tasks.  Despite 
the  potential  difficulty  of  talking  and  working  with  English  text  concurrently,  we  note  that 
think-aloud  procedures  have  been  efficaciously  used  with  ESL  students  in  previous  work  on 
reading  and  reading  strategies  (e.g.  Benedetto,  1986;  Block,  1986  a.b).  The  present  report 
focuses  soley  on  the  performance  characteristics  of  the  question  answering  task.  By  design,  the 
questions  varied  with  respect  to  the  degree  of  text  -  internal  versus  text  -  external  reasoning 
(Goldman,  1985)  required  for  correct  solution.  This  dimension  refers  to  the  degree  to  which  the 
text  provides  all  of  the  information  needed  to  answer  the  question,  as  compared  to  questions  that 
require  inferences  that  go  beyond  the  text  to  be  answered.  In  addition  to  the  variation  in  the  task 
demands  for  the  questions,  an  additional  level  of  variation  was  introduced  by  presenting  learners 
with  a  text  section  that  had  already  been  studied  in  class  (from  Chapter  2)  and  one  that  had  not 
been  studied  (from  Chapter  1 2)  at  the  time  we  tested  them.  Thus,  in  answering  questions  on  the 
'old"  material  (Chapter  2)  each  student  should  have  had  more  knowledge  than  when  answering  the 
questions  on  the  "new"  material. 

Method 

Materials. 


Two  sections  from  the  text  used  in  an  introductory  oceanography  course,  Ocean 
Science  (Stowe,  1983)  were  selected.  The  text  sections  were  approximately  1500  words  in 
length  and  featured  a  representative  sampling  of  the  type  of  concepts  students  in  the  course  have 
to  deal  with.  These  indude  definitions  of  terms,  properties  of  geophysical  phenomena,  and 
mathematical  relations  and  physical  laws.  The  first  section  was  from  Chapter  2  and  covered 
methods  for  studying  the  earth's  interior.  It  induded  one  formula,  one  table,  and  six  figures 
illustrating  various  geophysical  principles,  including  the  effects  of  subsurface  mass 


distributions  on  gravitational  force,  Snell's  law,  and  properties  of  sound  waves.  The  text  was 
parsed  into  157  predicate  propositions  (Kintsch,  1974)  or  idea  units  (Chafe,  1985).  These 
were  characterized  with  respect  to  intersentential  cohesion  and  types  of  logical  relations.  The 
second  section  was  from  Chapter  1 2  and  described  the  relationship  between  the  ocean  and  climate 
and  included  evaporation,  the  hydrologic  cycle  and  thermal  flow.  There  were  two  tables  and  one 
figure.  This  section  contained  154  predicate  propositions  and  employed  intersentential  cohesive 
relations  that  were  roughly  equivalent  to  those  found  in  Chapter  2.  Comprehension  questions 
were  drawn  from  the  study  guide  for  the  class  and  in  three  cases  were  developed  by  the  authors. 
Questions  varied  with  respect  to  the  envisionment  or  comprehension  level  necessary  to  arrive  at 
a  successful  answer.  (Appendix  A  contains  reproductions  of  the  text  selections,  the  questions  and 
their  answers.) 

Subjects. 

Seven  students  participated  in  the  question-answering  protocol  study.  Six  were 
enrolled  in  the  introductory  oceanography  course  and  the  seventh  (HS)  was  the  teaching  assistant 
for  the  course.  Students  completed  a  background  questionnaire  dealing  with  academic 
information,  language  skills  and  study  habits.  Table  1  provides  a  summary  of  the  most  pertinent 
information  from  this  questionnaire.  The  teaching  assistant  and  three  of  the  students  were  native 
English  speakers  (MR,  LH,  and  DW).  The  other  three  students  (GL,  II,  and  EH)  were  from  three 
different  non-English  language  backgrounds  but  were  relatively  proficient  in  English.  GL  rated 
his  English  language  skills  "extremely  good";  II  and  EH  each  rated  their  skills  "good".  The  four 
native  English  speakers  rated  their  own  English  language  skills  "extremely  good."  The  language  of 
instruction  during  high  school  had  been  English  for  ail  the  students.  EH  had  had  the  least 
exposure  to  English,  having  entered  the  United  States  six  years  ago,  at  which  time  she  had  her 
first  contacts  with  English.  II  and  GL  were  first  exposed  to  English  at  the  age  of  4  years.  Of  the 


non-native  English  speakers,  only  EH  reported  using  her  native  language  daily.  GL  and  II 
reported  that  they  used  their  native  languages  to  read  newspaper  or  magazine  material  but 
rarely. 1 

All  students’  high  school  backgrounds  included  courses  in  chemistry  and/or  biology. 

MR  and  DW  were  college  seniors,  GL  a  junior,  LH  a  sophomore  and  EH  and  II  were  freshmen  at  the 
time  of  the  study  and  experience  with  college-level  science  courses  varied.  MR  had  an  extensive 
background  in  physics  and  HS  in  oceanography.  DW  and  GL  had  taken  astronomy.  The  other  three 
students  had  no  college-level  science  courses  but  II  had  had  physics  in  high  school.  The  majors  of 
the  seven  students  varied  and  included  physics  (MR),  business/economics  (EH,  II), 
communications  (LH)  and  sociology  (DW).  Table  1  also  describes  the  self-ratings  of  English 
language  skills  related  to  academic  performance.  These  ratings  suggest  that  the  native  and 
nonnative  English  speakers  differ  primarily  in  terms  of  their  command  of  science  vocabulary. 

Ratings  on  learning  from  English  lectures  were  in  'good''  and  "extremely  good"  for  all  students. 
Learning  from  English  texts  skills  were  rated  higher  ("extremely  good")  by  the  native  English 
speakers  as  compared  to  the  nonnative  English  speakers  ("good"). 

Procedure. 

Students  were  recruited  through  the  introductory  oceanography  class.  They 
volunteered  to  participate  in  four,  2-hour  sessions  and  were  paid  $5  per  hour  for  their 
participation  at  the  conclusion  of  the  fourth  session.  Students  were  told  that  we  were  interested 
in  how  they  went  about  answering  questions  on  oceanography  material  and  that  we  wanted  them  to 
think  aloud  as  they  worked  on  several  questions  that  we  would  give  them.  Students  completed  the 
background  questionnaire  first.  Then  the  think-aloud  method  was  described  and  modeled, 
following  procedures  outlined  by  Ericsson  and  Simon  (1984).  Subjects  were  encouraged  to  talk 
aloud  as  they  were  working  each  question  and  were  told  that  they  would  be  prompted  to  verbalize 


if  there  were  periods  of  silence.  The  technique  was  illustrated  with  a  mental,  multicolumn 
multiplication  problem.  After  the  sut.  act  had  completed  the  think-aloud  solution,  instructions 
for  a  retrospective  talk*aloud  protocol  were  given:  subjects  were  instructed  to  report  what  they 
had  been  thinking  from  the  time  they  heard  the  question  until  the  time  they  gave  the  answer.  Two 
more  practice  exercises  were  given  ("How  many  windows  in  your  parents'  house?"  and  "Name  20 
animals")  and  subjects  provided  both  think  aloud  and  retrospective  protocols.  Note  that  although 
retrospective  training  was  given,  the  procedure  enacted  only  requested  a  retrospective  when  the 
think-aloud  protocol  was  very  brief  and  noninformative.  This  was  done  because  using  both 
became  too  repetitious  and  arduous  for  the  subjects. 

During  the  first  session,  each  student  was  given  the  "old"  selection  (the  section  from 
Chapter  2)  and  six  questions,  one  at  a  time.  After  the  student  had  answered  the  questions,  the 
experimenter  asked  subjects  to  go  back  over  the  text  selection  and  indicate  what  part(s)  were 
particularly  difficult  to  negotiate. 

Approximately  one  week  later,  each  subject  returned  for  the  second  session  on  Chapter 
2.  They  were  asked  to  orally  recall  what  they  remembered  from  the  selection  read  during  the 
first  session.  Then  they  were  given  the  next  two  subheaded  sections  of  Chapter  2  to  read  and 
outline,  employing  the  think-aloud  method  during  the  outlining. 

Approximately  three  to  four  weeks  later,  each  subject  returned  for  the  third  session. 
During  this  session,  the  first  section  from  Chapter  12  was  presented  and  subjects  answered  six 
questions  and  performed  a  difficulty  analysis.  Returning  one  week  later,  oral  recall  of  this 
section  and  reading  and  outlining  of  the  continuation  of  this  section  were  completed.  All  sessions 
were  audiotaped  and  later  transcribed.  This  paper  deals  only  with  the  question  answering  task. 


‘vVWW 


7 


Analysis  of  the  Question-answering  Protocols 


Protocol  analysis  was  guided  by  the  question  answering  model  shown  in  Table  2.  The 
model  was  developed  to  specifically  deal  with  questions  on  academic  text  material.  The  model 
indicates  four  major  processing  events  and  the  goals  associated  with  each  one.  In  addition, 
metacognitive  processing  events  are  shown  as  optional.  Monitoring  can  apply  to  any  of  the 
processing  events  and/or  to  learners’  general  thoughts  about  their  performance,  the  task,  the 


text,  etc. 


Each  major  processing  event  may  be  further  “unpacked"  or  expanded,  into  its 


constituent  processes,  goals  and  procedures.  The  first  processing  event,  question  encoding,  has 
two  primary  goals.  The  expansion  of  these  goats  is  shown  in  Table  3.  The  first  goal  is  to 
determine  the  type  of  question.  Questions  vary  in  terms  of  the  level  of  envisionment  demanded 
and  the  level  is,  in  part,  dependent  on  the  relationship  between  the  text  and  the  question. 
Processing  difficulty  and  the  amount  of  reasoning  required  for  successful  question  answering 
varies  with  such  question  requirements.  For  example  a  question  that  requires  an  explanation  that 


must  be  i 


the  material  given  in  the  text  will  require  more  processing  resources 


than  a  question  whose  answer  can  be  found  verbatim  in  the  text,  regardless  of  whether  it  is  an 
explanation,  comparison,  or  simple  "fact."  Two  example  questions  that  were  used  in  the  present 
study  illustrate  the  nature  of  the  differences  in  task  demands  among  the  question  set.  Question 
12,3  has  five  parts  that  vary  in  task  demands: 


a)  Suppose  air  at  25°  C  is  saturated  (100%  relative  humidity).  What  fraction  of 
the  air  is  water?  (b)  What  would  be  the  answer  if  the  temperature  were  35°? 
(c)  1 5°?  (d)  5°?  (e)  Does  the  answer  change  by  roughly  a  factor  of  2  for  every 
10°? 
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The  answers  to  parts  a  and  e  can  be  found  verbatim  in  the  text;  the  answers  to  c  and  d  can  be 

read  directly  from  a  table  given  in  the  text.  The  answer  to  b  must  be  computed  but  it  can  be 

computed  in  two  ways.  The  individual  can  either  apply  the  rule  referred  to  in  part  e  of  the 

question  or  can  interpolate  from  the  series  given  in  the  table  in  the  text.  Another  example, 

Question  2,6  illustrates  an  explanation  question  where  vocabulary  is  a  critical  issue:  "Briefly 

explain  how  'echoes'  can  be  used  to  measure  the  depths  of  discontinuities  in  the  earth's  internal 

structure."  To  locate  relevant  portions  of  the  text  or  to  access  the  appropriate  concept  in 

memory,  the  learner  must  understand  the  equivalence  between  the  phrase  in  the  question,  depths 

of  discontinuities ,  and  the  phrase  used  in  the  text  interfaces  between  materials.  Furthermore, 

there  are  several  sections  of  the  text  that  are  relevant  and  two  factors  that  need  to  be  discussed, 

time  and  speed.  Time  is  discussed  in  a  three  sentence  section  that  is  separated  by  nine  sentences 

from  the  section  that  deals  explicitly  with  the  time  and  speed  relationship.  To  give  the  complete 

answer,  the  learner  must  integrate  across  sections  of  text  and  extract  the  pertinent  relationship. 

The  text  does  indicate  that  the  learner  ought  to  read  on  for  the  complete  answer: 

"The  first  item  (time)  could  be  read  from  your  seismograph,  but  there  is  no 
direct  way  to  know  the  speed  of  seismic  waves  deep  within  the  earth. 

Fortunately,  this  information  may  be  inferred  from  data,  due  to  the  other 
important  property  of  waves. 

This  second  property  is...."  (speed  is  mentioned  again  3  sentences  later.) 

However,  not  all  learners  were  sensitive  to  this  cue  when  they  answered  this  question. 

To  accomplish  the  goal  of  determining  what  the  question  requires,  the  primary  means, 
as  shown  in  Table  3,  is  to  rely  on  the  language  of  the  question.  Analysis  of  the  language  draws  on 
prior  experiences  with  other  academic  texts  and  a  sensitivity  to  the  semantics  of  various  question 
words,  such  as  how  many,  what  and  why.  To  the  knowledgeable  individual,  these  questions  words 
provide  cues  to  the  appropriate  form  of  the  answer.  A  "how  many"  question  ought  to  indicate  the 
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need  for  a  specific  quantity,  a  "why"  for  a  causal  or  logical  explanation  of  the  phenomenon 
mentioned  in  the  question.  In  addition,  learners  may  use  the  task  context  and  constraints  to 
further  define  the  nature  of  the  answer.  In  an  untimed  situation,  one  can  afford  to  be  more 
discursive  than  in  a  time-limited  one.  Furthermore,  the  space  provided  for  the  answer  often 
provides  information  to  the  learner;  compare  for  example,  fill-in-the-blank  with  a  half  page  of 
blank  space. 

The  second  major  goal  of  question  encoding  is  to  determine  starting  points  for  a  search 
space.  A  primary  means  of  doing  this  is  to  use  technical  terms  and  keywords  concepts  mentioned 
in  the  question  as  entry  points  to  memory  and/or  the  textbook.  When  the  question  uses  words  that 
match  those  used  in  the  headings  and  subheadings  of  the  text,  learners  are  virtually  assured  of  a 
reasonably  well-defined  search  space  in  the  text.  Key  term  matching  is  also  facilitated  by  the 
use  of  boldface  or  italics  in  the  body  of  the  text.  In  defining  a  search  space  in  memory,  the  key 
terms  in  the  question  behave  similarly  but  the  success  of  a  match  will  vary  depending  on 
learners'  individual  mental  representations  of  the  text  information.  Regardless  of  whether  the 
learner  is  defining  a  search  space  in  the  text  or  in  semantic  memory,  there  is  the  possibility  that 
the  words  in  the  question  will  be  seductively  appealing  as  cues,  when  in  fact  the  question  really 
requires  original  thinking  or  the  application  of  presented  material.  Questions  of  this  sort  are  not 
only  at  more  complex  envisionment  levels  but  also  provide  "false"  signals  to  the  learner, 
creating  the  impression  that  the  answer  is  "in  the  book". 

As  indicated  above,  an  important  outcome  of  question  encoding  is  the  definition  of  the 
search  space  and  type  of  answer  required.  Search  proceeds  either  in  memory  or  in  an  external 
source  such  as  a  textbook,  notes  or  supplementary  reference  material.  The  goals  differ  somewhat 
for  memory  and  text  searches,  as  shown  in  Table  2.  The  goals  for  memory  search  distinguish 
between  two  outcomes.  In  the  first  goal,  the  answer  is  found  in  memory  and  no  external  search  is 


v  « 
v\ 

>>  l> 


yCy 

yv 

-v 


'.V, 

n 

>2 


» 


»\v 


» 


» 


f.V 


• 


y. 


aV. 


undertaken.  For  this  to  occur,  we  assume  that  decision  criteria  operate  and  that  any  memory 
search  is  monitored  and  its  progress  evaluated.  We  further  assume  the  existence  of  a  threshold 
or  criterion  against  which  candidate  answers  are  tested.  When  an  answer  exceeds  threshold  it  is 
output  by  the  learner.  Memory  search  that  does  not  produce  an  answer  exceeding  critical  value 
for  output  will  eventuate  in  two  outcomes:  (1)  the  learner  will  conclude  that  the  answer  is  not 
known,  or  cannot  be  remembered:  (2)  the  learner  will  conclude  that  external  sources  are  needed 
to  find  the  answer.  Individual  differences  in  learners  govern  the  extensiveness  of,  and 
persistence  in,  memory  searches.  If  memory  search  is  abandoned  for  external  source  search,  a 
second  goal  of  the  memory  search  is  to  determine  alternate  sources  for  the  answer.  Thus,  a 
memory  search  may  not  produce  the  answer  but  may  further  define,  and  refine,  subsequent 
search  spaces.  External  source  search  can  thus  be  facilitated  by  a  memory  search  that  does  not 
yield  the  specific  answer  to  the  question.  The  memory  search  phase  of  the  model  is  the  one  about 
which  we  will  have  the  least  to  say  based  on  the  protocol  data  (see  Reder,  1 987,  for  a  recent 
theoretical  and  empirical  focus  on  question  answering  from  memory). 

Goals  for  external  searches  are  similar  to  those  for  memory  in  that  we  assume  the 
operation  of  decision,  monitoring  and  evaluation  processes  during  the  course  of  the  search.  In 
searching  a  textbook,  learners  may  have  three  major  goals  that  are  interdependent  and 
interrelated,  as  outlined  in  Table  4.  One  goal  is  to  delimit  the  search  space  by  using  the  results  of 
question  encoding  and  memory  search  to  constrain  and  guide  textbook  processing.  This  goal 
becomes  increasingly  important  as  the  amount  of  potentially  appropriate  text  material  increases. 

It  would  be  relatively  unimportant  when  the  student  knows  that  the  questions  pertain  to  a  section 
consisting  of  only  a  few  pages  of  text. 

A  second  goal,  finding  information  relevant  to  the  question, witl  be  achieved  with  less 
effort  if  the  search  space  can  be  appropriately  limited  but  will  be  more  difficult  if  the  search 


space  is  incorrectly  constrained.  Given  a  section  of  text  to  search,  the  search  may  be  global  or 
guided,  as  shown  in  Table  4.  Global  searches  commence  with  the  learner  having  only  a  vaguely 
defined  search  space,  e.  g..  Til  look  in  the  text  for  that.  It  must  be  there  somewhere."  Such 
searches  are  typically  characterized  by  scanning  or  skimming  of  text  and  may  be  exhaustive  or 
self-terminating.  Monitoring,  evaluation  and  decision  processes  are  tuned  to  the  occurrence  of 
concepts,  vocabulary,  or  other  text  material  that  "matches"  the  requirements  of  the  question. 

When  a  match  is  encountered  the  learner's  attention  becomes  focused  on  that  section  of  text  and 
more  careful  examination  of  the  text  replaces  the  skimming  behavior.  The  search  process,  in 
effect,  changes  to  a  guided  search.  Guided  search  is,  from  the  outset,  targeted  at  a  defined  search 
space,  a  localized  area  of  the  text.  Key  words  and  topics  mentioned  in  the  question  are  used  by  the 
learner  to  explicitly  identify  and  focus  on  specific  sections  of  text,  e.  g.,  "This  question  is  about 
gravity.  That’s  section  two.  I’ll  look  there." 

Whether  learners  employ  guided  or  global  search  strategies  is,  in  part,  related  to  the 
degree  to  which  the  search  space  has  been  delimited  and  the  question  appropriately  encoded. 
Frequent  monitoring  of  the  utility  of  text  searches  is  important  for  efficient  and  successful 
search  behavior.  In  particular,  as  Table  4  indicates,  when  the  question  requires  that  text  be 
paraphrased,  summarized  or  be  the  input  to  some  sort  of  reasoning  and  analysis  process, 
evaluation  of  the  pertinence  of  the  specific  text  "facts"  is  essential  to  the  learner's  success. 

Failure  of  this  process  can  lead  to  extended  and  unsuccessful  text  searches.  The  converse  is  true 
regarding  false  recognition  of  text  information,  that  is,  recognizing  irrelevant  information  as 
relevant  to  the  answer. 

Once  relevant  information  is  located  it  must  be  meaningfully  processed.  The  learner 
must  extract  the  relevant  information  in  a  form  that  suits  the  task  demands.  Example  means  to 
accomplish  this  are  listed  in  Table  4.  For  some  questions,  just  recognizing  the  right  material  and 


reading  it  from  the  text  is  sufficient.  For  other  types  of  questions,  summarization  may  be  called 
for  and  in  still  others,  the  learner  may  need  to  engage  in  extended  reasoning  and  knowledge 
application  processes. 

DW*s  protocol  for  question  6  from  chapter  12  that  required  the  explanation  of  an 
everyday  occurrence  illustrates  the  interdependence  and  interrelatedness  of  the  goals  and 
processing  events  in  the  question  encoding,  memory  search  and  text  search  components  of  the 
model.  This  particular  question,  "Explain  why  you  feel  cold  when  you  get  out  of  the  shower",  is 
directly  answered  in  the  text  and  the  specific  information  is  contained  in  one  paragraph. 

However,  references  to  the  concept  involved,  "evaporation",  occur  in  the  three  paragraphs 
preceding  the  answer  and  in  the  four  paragraphs  following  the  answer.  DWs  coded  protocol  is 
shown  in  Table  5.  She  first  encoded  the  question  by  reading  it  and  defining  the  topic  as  one  of  the 
major  ones  in  that  section  of  text,  "latent  heat".  She  then  gave  an  answer  from  memory,  "because 
of  the  evaporation  process’,  but  evaluated  her  answer  as  insufficient  for  the  question.  She 
proceeded  to  a  text  search,  presumably  to  gather  more  information  on  evaporation  and  why  the 
process  works  that  way.  She  began  reading  in  the  appropriate  section,  B.2  Latent  heat,  but  two 
paragraphs  below  the  one  containing  the  answer.  She  skipped  up  the  page  to  the  topic  sentence  of 
the  paragraph  containing  the  answer  (B.2S15).  She  then  stated  an  answer  that  paraphrased  Si  5 
in  nontechnical  terms,  deleting  the  details  of  molecular  movement  and  its  relationship  to 
evaporation  and  temperature.  She  evaluated  this  answer  as  missing  something  and  adopted  a 
global  search  strategy  of  skimming  from  the  beginning  of  Chapter  12  until  she  got  to  the  first  two 
sentences  of  section  B.1,  which  use  the  key  term  "evaporate."  She  read  two  sentences  and  then 
skimmed  the  remainder  of  section  B.1 ,  assumedly  because  the  "evaporation"  match  didn’t  lead 
anywhere  on  the  shower  explanation.  She  then  read  the  first  six  sentences  under  section  B.2,  and 
recognized  this  as  the  relevant  material.  She  re-read  part  of  it  (S3  -  S6)  and  then  continued 
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reading  about  perspiration  and  evaporation.  This  is  also  why  we  perspire.  (S7)  The  heat 
required  to  evaporate  the  water  comes  from  our  skins,  cooling  us  off.  (S8)"  DW  then  proceeded 
to  paraphrase  the  information  she  had  just  read.  However,  she  got  tangled  up,  reread  S8  and  then 
reverted  to  her  original  answer: 


Well,  I  guess  I'll  do  it  (answer)  in  sort  of  a  round  about  way.  First  of  all,  the 
heat,  the  hot  water  of  the  shower.  You  get  out  of  the  shower,  and  the  water,  which 
is  storing  the  heat  on  your  body,  is...You  usually  dry  it  off,  and  the  evaporation  ... 
Urn  ...  Actually,  that's  not  true  'cause  if  it's  cold  outside  the  evaporation  is  a 
slower  process.  The  heat  required  to  evaporate  the  water  comes  from  our  skin." 
Well,  I  guess  I'd  say  evaporation,  and  leave  it  at  that. 


DW  started  by  defining  the  search  space  as  information  on  latent  heat;  she  then  retrieved  an 
answer  from  memory,  evaporation.  This  answer  guided  her  attempts  to  search  the  text  for  more 
information.  However,  DW  failed  to  see  the  importance  of  continuing  with  the  paragraph  starting 
with  SI 5;  instead  she  engaged  in  a  global  text  search  that  took  her  away  from  the  relevant 
information.  She  never  got  back  to  this  paragraph  but  attempted  to  conclude  her  answer  to  this 
question  with  a  paraphrases  of  a  less-relevant  text  section  (B.2S7  &  8).  However  this  section  is 
the  first  section  under  her  self-defined  topic,  “latent  heat,’  that  mentioned  the  key  term  in  her 
answer,  "evaporation."  She  monitored  her  understanding  and  the  paraphrase  as  not  terribly 
direct  but  instead  of  resuming  a  search  process,  she  retreated  back  to  the  same  answer  she  had 
originally  retrieved  from  memory  and  the  one  that  provided  the  impetus  for  the  initial  text 
search.  We  can  only  speculate  on  why  she  "gave  up’  at  this  point. 

DW*s  protocol  also  illustrated  the  fourth  component  of  the  question  answering  model,  as 
shown  in  Table  2:  Construct  and  output  an  answer.  Two  goals  involved  here  are  to  answer  the 
question  completely  and  match  the  type  of  answer  to  the  type  of  question.  Question  1 2,6  required 
an  explanation  and  DW  indicated  to  us  that  her  initial  answer  did  not  really  qualify  as  an 
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explanation  by  her  self-question  'But  what  is  it  that  causes  that?"  Another  learner,  GL, 
indicated  his  attempt  to  meet  the  goal  of  answering  completely  after  giving  a  correct  partial 
answer  to  question  2,6.  He  stated:  "Ok.  I'll  read  it  again  and  to  find  out  exactly  what  these  are 
used  for.  I  mean  I  know  they’re  used  to  measure  depth  but  I  know  they  have  other  (uses)."  Table 
6  "unpacks"  the  two  aforementioned  goals  of  answering  and  indicates  a  third,  optional  goal  of 
demonstrating  that  the  information  has  been  incorporated  into  the  knowledge  base.  Several  of  the 
students  repeatedly  paraphrased  text  sections  they  had  just  read  aloud.  We  interpret  this 
behavior  as  their  efforts  to  comply  with  this  optional  goal.  Attention  to  this  goal  may  be  one  of 
the  distinguishing  features  of  expert  learners. 

Finally,  the  question-answering  model  features  optional  processes  that  relate  to  the 
role  of  metacognitive  behavior  in  the  question  answering  process.  At  any  time,  and  for  any  of  the 
"processing  events",  confirmation,  monitoring  and  evaluation  may  be  invoked  by  the  learner.  The 
outcomes  of  such  monitoring,  in  part,  determine  the  sequencing  and  interplay  between  the 
various  processing  events.  For  example,  after  reading  a  lengthy  section  of  the  text,  a  number  of 
students  were  observed  to  reread  the  question.  There  are  several  related  explanations  for  why 
that  particular  sequence  occurred:  first,  the  learner  might  have  monitored  memory  for  the 
specifics  of  the  question  and  determined  that  the  trace  was  not  sufficiently  active;  second,  the 
juxtaposition  of  the  text  and  the  question  might  help  in  the  determination  of  the  appropriateness 
of  the  just-read  text  to  the  exact  question.  Reder  (1 987)  has  suggested  a  third  possibility: 
rereading  may  increase  the  familiarity  of  the  concepts  and  thus  the  likelihood  of  success  for 
direct  retrieval  of  the  answer. 


The  question  answering  model  leads  to  a  number  of  solution  strategies  depending  on  the 
particular  selection  of  processing  events,  the  specific  goals  that  learners  seek  to  meet  within 
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each  processing  event,  and  the  degree  and  type  of  involvement  of  monitoring  processes.  A 
typology  of  solution  strategies  is  shown  in  Table  7.  A  strategy  consists  of  question  encoding,  some 
type  of  search,  some  set  of  additional  processing  events  (including  the  empty  set)  and  output  of 
the  answer. 

Four  types  of  search  are  specified,  each  of  which  can  be  accompanied  by  any  of  four 
"additional"  processing  events:  question  analysis,  reasoning/inference,  process  monitoring  or 
product  monitoring.  Types  A  -  D  differ  in  terms  of  the  types(s)  of  searches  that  are  undertaken 
and  the  outcome(s)  of  the  search.  Search  Types  A,  B  and  C  involve  searches  for  which  a 
successful  outcome  is  possible,  i.e.  the  correct  answer  can  be  “located"  in  memory  (A),  in  the 
text  (B)  or  by  employing  a  combination  of  text  and  memory  searches  (C).  In  contrast,  Type  D 
represents  a  situation  where  the  information  is  Q£l  in  the  text  or  could  not  have  been  stored  in 
memory  unless  the  learner  had  solved  the  problem  previously.  Type  D  searches  lead  to  the 
recognition  of  the  need  to  uSfl  given  information,  either  by  applying  some  rule  or  formula  given 
in  the  text  or  by  using  the  information  in  a  novel  problem  solving  context. 

Each  search  type  may  be  augmented  by  the  occurrence  of  one  or  more  of  the  four 
additional  processing  events.  Question  analysis  refers  to  rereading  or  analyzing  the  nature  of  the 
question.  Reasoning  and  inference  refer  to  efforts  to  logically  manipulate  given  or  remembered 
information.  Process  and  product  monitoring  refer  to  metacognitive  behavior  directed  at  the 
progress  of  any  of  the  processing  events  (process  monitoring)  or  at  results  of  search  and 
retrieval  processes  (product  monitoring).  The  latter  include  reality-testing  and  checking 
behaviors  directed  at  answers  or  candidate  answers.  Process  monitoring  includes 
self-questioning  aimed  at  the  text,  the  question  or  the  learner's  own  internal  state  during  the 
problem  solving  process.  These  are  often  evaluations  of  the  relevance  of  specific  portions  of  text, 
of  whether  or  not  a  particular  concept  sounds  familiar,  or  of  whether  a  new  means  might  be 
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appropriate.  Both  types  of  monitoring  are  regulatory,  and  typically  result  in  decisions  regarding 
the  solution  process. 


Protocol  Scoring. 

Each  protocol  was  analyzed  in  terms  of  the  processing  events  that  have  been  outlined  in 
the  discussion  of  the  question  answering  model.  Specifically,  individual  actions  were  classified 
according  to  cognitive  actions  (recall,  read,  compare,  monitor,  evaluate)  and  information  on 
which  the  event  acted  (text  material,  question,  own  knowledge).  A  full  set  of  the  cognitive  actions 
used  in  coding  is  provided  in  Appendix  B.  Table  5,  DW’s  protocol  for  question  12,  6,  shows  an 
example  of  the  coded  protocols  that  resulted  from  this  procedure. 

The  next  step  in  the  data  reduction  process  was  to  assign  each  coded  question  answering 
protocol  to  a  solution  strategy  type  by  indicating  the  search  type  and  the  presence  or  absence  of 
each  of  the  "additional”  processes.  We  adopted  the  convention  of  a  5-place  code  for  solution 
strategy  types,  where  the  First  place  indicated  the  type  of  search(A  -  D)  and  the  remaining  four 
indicated  the  presence  (1 )  or  absence  (0)  of  each  of  the  processing  events  named  in  the  columns 
in  Table  7.  Thus,  the  code  B.1010  indicates  a  strategy  involving  text  search,  question  analysis 
and  process  monitoring.  Furthermore,  a  superscript  gives  the  evaluation  of  the  final  answer 
produced  by  the  learner,  with  1  indicating  correct,  2  indicating  qualitatively  correct  (but  not 
quantitatively)  for  those  questions  where  a  specific  numerical  value  was  requested,  3  indicating  a 
partially  correct  answer,  4  an  incorrect  answer  and  5  no  answer  given. 

Question  types 

The  solution  strategy  employed  by  the  learner  is  a  function  of  the  question,  the  text  and 
the  learner's  knowledge  base.  Five  question  types  were  identified  based  on  an  analysis  of  (1)  the 
relationship  between  the  question  and  text  and  (2)  on  the  demands  made  on  the  knowledge  base. 
Table  8  summarizes  the  five  types  and  indicates  how  the  18  separate  questions  employed  in  the 


study  were  distributed  across  types. ^  In  the  first  type  of  question  there  is  a  verbatim 
relationship  between  the  question  and  the  text.  The  text  gives  the  answer  explicitly  and  there  is  a 
direct  match  between  the  question  wording  and  the  text  wording.  Questions  of  this  type  may  ask 
for  quantitative  answers,  comparisons,  explanations,  conclusions,  etc.  In  all  cases,  however,  the 
learner  merely  has  to  locate  the  appropriate  section  in  the  text  and  find  the  matching  language.  In 
the  particular  sample  of  questions  studied  here,  there  Were  5  instances  of  this  type  of  question; 
for  3  of  them,  quantitative  responses  were  appropriate,  1  required  a  comparison  between 
quantitites  and  1  asked  for  the  properties  of  a  concept. 

The  second  type  of  question  involves  a  paraphrase  relationship  between  the  question 
and  the  text.  The  text  gives  the  answer  explicitly  if  vocabulary  equivalences  and  conversions  are 
understood.  Questions  may  vary  with  respect  to  the  degree  of  conversion  needed.  Sometimes  only 
one  or  two  words  might  differ  between  the  text  and  the  question;  other  times  the  majority  of  the 
question  wording  might  differ  from  that  of  the  text.  These  conversions  frequently  depend  on 
prior  knowledge  and  assume  that  the  learner  already  knows  certain  technical  vocabulary  (e.g. 
factor,  ratio).  Typically,  however,  once  the  appropriate  vocabulary  conversions  are  done  and 
conceptual  equivalences  are  established,  the  relevant  text  portion  is  relatively  circumscribed 
and  localized  in  one  portion  of  the  text.  As  with  the  first  type  of  question,  answers  to  these 
questions  may  involve  quantitative  responses,  explanations,  definitions,  etc.  There  were  four 
questions  of  this  type,  one  of  which  required  a  quantitative  response  and  three  of  which  required 
explanations  (Why?,  How  do  we...,  and  How  can...). 

The  third  type  of  question,  verbatim  look-up  plus  comparison,  involves  reasoning 
with  information  found  in  the  text.  The  text  gives  the  necessary  information  explicitly  and  there 
is  usually  a  direct  match  between  the  question  wording  and  the  text.  Thus,  locating  the 
information  proceeds  much  like  in  type  1  questions.  However,  once  the  information  is  "found",  it 


must  then  be  compared  to  other  information.  Sometimes  concepts  must  be  compared;  other  times 
quantities.  In  many  questions  of  this  type,  a  table  or  figure  in  the  text  will  contain  the 
information.  There  were  three  instances  of  this  type  of  question  in  the  oceanography  sample,  two 
of  which  involved  locating  and  then  comparing  quantities;  the  third  involved  a  comparison  of  two 
concepts. 

The  fourth  type  of  question  requires  integration  of  information  across  several 
paragraphs  of  the  text.  There  may  be  either  a  verbatim  or  paraphrase  relationship  between  the 
question  and  the  text  but  the  text  provides  the  relevant  information  in  a  number  of  paragraphs. 

The  information  must  be  coordinated  and  analyzed  to  construct  the  correct  answer.  These  types  of 
questions  would  lend  themselves  to  partially  correct  answers  if  learners  were  to  locate  only  one 
relevant  section  of  text.  In  the  oceanography  sample  there  were  4  instances  of  this  type  of 
question.  Three  required  explanations  of  concepts  or  processes  and  one  requested  the  difference 
between  two  concepts.  In  all  four,  there  was  a  verbatim  relationship  between  the  text  and  the 
question,  although  the  matching  wording  occurred  in  several  paragraphs  of  the  text. 

The  last  type  of  question  requires  reasoning,  application  and/or  computation. 

Questions  of  this  type  involve  using  a  text  provided  formula,  rule  or  relationship  to  get  the 
precisely  correct  answer.  Locating  the  formula,  rule  or  relationship  involves  a  verbatim  or 
paraphrase  match  or  look  up  process.  Answers  to  questions  of  this  type  cannot  be  found  in  the 
book  directly.  Rather,  the  learner  must  disembed  the  relevant  information  and  apply  it  to  a  new 
situation  described  in  the  question.  This  type  of  question  thus  requires  envisionment  or 
comprehension  levels  that  involve  extending  the  text  beyond  its  own  confines  whereas  the  other 
types  of  questions  typically  stay  within  localized  portions  of  the  text.  In  the  oceanography 
sample,  there  were  two  examples  of  this  question  type.  In  the  first,  a  quantitative  response  was 
to  be  determined  using  a  formula  provided  in  the  text.  However  the  symbols  in  the  formula  and 


their  text-provided  explanations  did  not  directly  match  the  terms  given  in  the  question.  To  make 
the  correct  equivalences,  prior  knowledge  of  the  abbreviations  used  in  the  formula  was 
necessary.  This  question  thus  also  had  elements  of  a  type  2  question.  The  other  question  of  type  5 
also  required  a  quantitative  response.  The  learner  had  to  determine  the  "next*  value  in  a  series 
that  appeared  in  a  table  in  the  text.  Thus,  using  the  tabled  values,  one  could  determine  the  correct 
answer.  Alternatively,  the  rule  governing  the  entries  in  the  table  was  given  in  the  text  and  one 
could  apply  that  rule  directly  to  get  the  answer. 

Predictions  regarding  the  protocols 

There  is  an  expected  ordering  of  difficulty  for  the  five  types  of  questions,  with  type  1 
expected  to  be  the  easiest  and  type  5  the  most  difficult.  Types  2, 3,  and  4  were  predicted  to  be 
roughly  equivalent  in  difficulty.  We  also  expected,  based  on  the  notion  that  the  need  for  various 
processing  events  depends  on  the  relation  between  the  question  and  the  text,  solution  strategies  to 
reflect  variations  in  task  demands  created  by  question  factors  (see  also  Reder,  1 987, 

Experiment  6).  Specifically,  reasoning/inference  and  question  analysis  were  predicted  to  occur 
less  often  for  Type  1  questions  as  compared  to  the  other  types.  We  predicted  that  monitoring 
processess  would  occur  more  often  for  questions  requiring  multiple-source  coordination,  e.g., 
types  3, 4,  and  possibly  5. 

Results  and  Discussion 

Several  important  trends  are  reflected  in  the  solution  strategy  data  shown  in  Table  9. 
Consistent  with  an  interactive  comprehension  model,  task  characteristics  as  reflected  in  the  type 
of  question,  and  learner  characteristics,  as  reflected  in  domain  expertise  and  in  language 
proficiency,  affected  the  nature  of  the  solution  strategies.  The  various  types  of  questions  were 
differentially  difficult  and  there  was  evidence  of  the  use  of  different  strategies  depending  on  the 
type  of  question .2  Furthermore,  individuals  tended  to  be  relatively  consistent  in  their  strategic 


approach  to  the  task  but  individual  differences  were  observed  and  were  related  to  both  domain 
expertise  and  to  language  background.  Each  of  these  trends  is  discussed  below. 


Task  difficulty  and  question  woe 

Question  type  difficulty  was  measured  by  the  percent  correct  (or  qualitatively 
correct)  answers  and  is  shown  in  the  first  row  of  Table  10.  As  predicted,  Type  1  questions  were 
the  easiest:  89%  of  the  answers  were  correct.  Type  4  questions  were  also  relatively  easy,  71% 
correct.  The  unexpected  ease  with  which  these  were  solved  is  probably  due  to  the  verbatim 
relation  that  held  between  this  sample  of  type  4  questions  and  the  text.  However,  another  sample 
of  type  4  questions  might  involve  vocabulary  conversion  as  well  as  cross-paragraph  integration, 
in  which  case  they  would  probably  be  harder  to  solve.  In  this  particular  task  context  the 
difference  between  types  1  and  4  was  the  amount  of  text  that  matched  the  question,  with  type  4 
questions  matching  over  longer  segments.  Those  question  types  requiring  reasoning  were 
successfully  solved  62%  (type  3)  and  57%  (type  5)  of  the  time,  whereas  type  2  questions  were 
the  most  difficult,  39%  successful  solutions.  If  one  considers  only  successful  quantitative 
solutions  for  type  5,  36%  of  the  solutions  were  correct.  Thus,  the  most  difficult  question  type, 
type  2,  was  the  type  requiring  knowledge  of  vocabulary  and  conceptual  equivalence,  primarily 
for  technical,  natural  science  and  oceanography  terms.  Note  that  correct  quantitative  solutions 
for  Type  5  questions  also  depended  having  on  this  type  of  knowledge  (especially  Chapter  2,  2)  and 
the  success  rate  was  similarly  low. 

Solution  strategies  and  question  tvoe 

Learners  engaged  in  different  strategies  depending  on  the  type  of  question  they  were 
attempting  to  answer.  For  the  type  1  questions,  verbatim  relation  to  text,  90%  of  the  correct 
solutions  involved  a  simple  "search  plus  retrieve  answer’  strategy.  Of  these  ,  58%  were  text 
searches  and  32%  were  memory  searches.  Furthermore,  half  of  these  single  source  searches 
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involved  no  other  processing  events.  Correct  solutions  to  type  4  questions,  which  also  featured 
verbatim  relations  to  the  text,  were  simple  "search  plus  retrieve  answer"  strategies  85%  of  the 
time,  but  about  half  of  these  were  text  and  half  memory  searches.  The  remaining  1 5%  used  both 
text  and  memory  searches.  Solution  to  the  two  types  of  verbatim  questions  differed  principally  in 
terms  of  the  more  frequent  use  of  monitoring  in  the  type  4  questions.  In  particular,  for  type  4 
questions,  65%  of  the  correct  solutions  (79%  of  ail  solutions)  included  process  and/or  product 
monitoring  compared  to  42%  for  Type  1  questions.  Not  surprisingly  reasoning  and  inference 
processes  rarely  occurred  for  either  type  1  or  4  questions.  These  data  are  summarized  in  Table 
10. 

Type  3  questions,  verbatim  "look  up"  plus  comparison,  were  similar  to  type  1  and  4 
questions  in  terms  of  the  search  strategies  (85%  search  single  source,  1 5%  search  both  text  and 
memory)  used  in  correct  solutions.  Type  3  differed  from  types  1  and  4  in  that  54%  of  the 
correct  solutions  employed  reasoning  and  inference  processes,  a  difference  consistent  with  the 
nature  of  these  questions.  Type  3  differed  from  type  4  but  was  similar  to  type  1  in  that  46%  of 
the  correct  solutions  featured  monitoring,  principally  checking  product  or  answer  before 
stopping.  Only  23%  of  the  correct  solutions  (24%  of  all  solutions)  failed  to  include  any  of  the 
four  processing  events  beyond  search  and  retrieval.  Thus,  for  these  three  types  of  questions  that 
all  in  some  measure  involve  verbatim  relations  between  text  and  question,  there  is  a  comparable 
degree  of  reliance  on  text  and  memory  search  leading  to  correct  answer  retrieval.  Types  1  and  4 
are  similar  to  one  another  but  different  from  type  3  on  the  use  of  reasoning;  type  1  and  3  include 
less  monitoring  than  type  4. 

Correct  solutions  to  type  5  questions,  requiring  use  of  a  formula  or  rule  given  in  the 
text,  are  distinguished  from  types  1, 4  and  3  in  terms  of  the  appearance  of  the  disaffirming 
search  strategy,  i.e.,  the  search  leads  to  the  realization  that  the  answer  is  not  "given"  explicitly 


in  the  text  but  must  be  computed  (search  type  D).  In  conjunction  with  this  search  type,  process 
and  product  monitoring  are  very  frequent  in  the  correct  solutions  and  are  more  frequent  than  in 
the  three  preceding  types  of  questions.  Similar  to  type  3,  reasoning  and  inference  occurred  in 
63%  of  the  correct  solutions.  Solutions  to  type  5  questions  rarely  (13%)  featured  no  additional 
processes,  a  marked  difference  from  types  1  and  4. 

Correct  solutions  to  type  2  questions  were  a  cross  between  type  1  and  4  solution 
strategies.  Like  type  1 , 83%  of  the  correct  solutions  involved  searching  one  source  and 
answering;  like  type  4,  there  was  a  high  degree  of  monitoring  (75%  of  the  solutions).  Note 
however,  that  these  data  represent  only  39%  of  the  attempted  solutions.  When  correct,  the 
successful  vocabulary  conversion  allowed  learners  to  treat  these  questions  like  verbatim 
questions.  Given  the  high  failure  rate  on  Type  2  questions  (61%),  consideration  of  the  solution 
strategies  for  both  correct  and  incorrect  final  answers  is  informative.  Over  all  solutions,  the 
most  frequent  search  strategy  was  a  text  search  (17  of  28  solutions);  however,  59%  (10)  of 
these  failed  to  lead  to  correct  solution.  Another  five  memory  search  solutions  (63%  of  8 
memory  searches  attempted)  failed  to  lead  to  a  correct  solution.  Two  of  three  solutions  using  both 
sources  failed.  Despite  the  fact  that  the  final  answers  were  incorrect,  learners  did  engage  in 
question  analysis  (71%  of  the  "incorrect*  solutions)  and  monitoring  (76%  of  the  "incorrect" 
solutions).  The  inability  to  map  the  language  of  the  question  onto  the  language  of  the  text  appears 
to  have  been  the  critical  obstacle  to  successful  solution.  We  note  that  nonnative  English  and 
native  English  speakers’  solutions  resulted  in  correct  answers  equally  often  on  this  question  type 
(42%  and  38%,  respectively).  However,  the  solution  attempts  of  the  former  group  were  more 
likely  to  be  incorrect  than  those  of  the  latter  (50%  versus  25%).  The  native  English  speakers' 
solutions  resulted  in  partially  correct  answers  31%  of  the  time  but  there  were  no  partially 
correct  answers  for  the  nonnative  English  speakers.  Thus,  difficulties  on  this  type  of  question 


may  be  related  to  English  language  proficiency.  In  addition,  they  are  also  substantially  related  to 
subject  matter  expertise  differences  and  the  impact  of  this  factor  on  the  solution  strategies  of 
individual  learners. 


The  learners  examined  in  this  study  differed  along  several  dimensions  other  than  native 
language.  In  particular,  and  as  noted  in  discussing  Table  1 ,  the  hard  science  backgrounds  of  two  of 
the  native  English  speakers  (MR  and  HS)  qualified  them  as  experts,  or  near  experts,  when 
dealing  with  the  material  in  Chapters  2  and  12  of  the  Oceanography  text.  Their  strategies  for 
answering  the  questions  differed  from  the  other  "novice"  native  English  speakers  and  from  the 
nonnative  English  speakers.  However,  one  nonnative  English  speaker  (GL)  who  had  taken  a 
science  course  in  college  (astronomy)  also  differed  in  approach  from  the  two  other  nonnative 
English  speakers  (EH  and  II)  and  from  the  novice  native  English  speakers  (DW  and  LH).  Table 
1 1  presents  the  quantifiable  characteristics  of  the  solution  strategies  for  each  learner. 

Considering  first  the  two  domain  experts,  MR  and  HS,  we  note  a  relatively  strong 
reliance  on  memory  searches,  relatively  little  question  analysis,  and  reasoning/inference  in 
about  one-third  of  the  solutions.  Both  also  engaged  in  monitoring  on  about  half  the  solutions.  The 
primary  difference  between  MR  and  HS  was  in  the  rate  of  correct  solutions:  MR  was  correct  89% 
of  the  solutions  compared  to  67%  for  HS.  This  difference  may  be  related  to  a  difference  in  the 
type  of  monitoring  done  by  MR  -  largley  checking  to  be  sure  the  answer  was  complete  -- 
compared  to  that  used  by  HS  --  largely  keeping  track  of  the  progress  of  her  efforts  to  solve  a 
particular  problem.  Thus,  MR's  solution  strategies  included  a  preponderance  of  successful 
memory  searches,  answer  confirmation  processes  and  reasoning  where  appropriate,  e.g.,  for 
question  types  3  and  5.  HS  employed  similar  strategies,  however  she  failed  to  monitor  her 
answers  very  well  and  the  outcomes  of  her  efforts  were  less  often  accurate.  Examining  her 
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protocols  revealed  a  tendency  to  quit  once  she  had  produced  an  answer  that  met  her  acceptability 
criterion.  As  the  TA  for  the  course,  HS  may  have  been  assuming,  perhaps  incorrectly,  that  she 
already  knew  many  of  the  answers  and  thus,  consulted  the  question  and  the  text  far  less  often  than 
many  of  the  other  learners. 

In  contrast  to  HS,  but  like  MR,  GL  had  a  high  rate  of  successful  solution.  However, 
compared  to  both  MR  and  HS,  GL  relied  almost  exclusively  on  text  searches  in  his  solutions. 
Furthermore,  he  engaged  in  frequent  question  analysis  and  monitoring  (about  50%  of  the 
solutions).  GL  monitored  both  process  and  product.  Two  of  the  three  questions  that  GL  got  wrong 
involved  vocabulary  conversion  or  paraphrase  to  successfully  map  the  question  to  the  text.  On  the 
whole,  however,  GL  was  generally  highly  successful  in  his  use  of  the  text. 

The  remaining  four  learners  tended  to  be  correct  less  often  than  MR,  GL,  or  HS.  LH  was 
correct  least  often  (44%  of  her  solutions).  Her  solution  strategies  were  characterized  by 
memory  search  or  text  search  but  little  monitoring  of  either  the  process  or  products  of  search. 

She  engaged  in  a  relatively  high  degree  of  question  analysis  and  an  average  amount  of  reasoning 
and  inference.  LH  appeared  to  engage  in  no  cognitive  activities  that  would  have  permitted  her  to 
reject  or  repair  inadequate  candidate  answers  and  had  the  highest  number  of  partially  correct 
answers.  The  other  "novices"  each  engaged  in  monitoring  activities  more  frequently  than  did  LH 
and  were  correct  on  55%  to  65%  of  their  solutions.  Of  the  three,  II  was  the  only  one  who  had 
taken  any  physics  courses.  IPs  solution  strategies  reflected  a  cross  between  MR  and  GL  with 
respect  to  search  types.  He  tended  to  engage  in  more  question  analysis  than  MR  and  about  the  same 
degree  of  monitoring  as  GL  and  MR.  Il's  lower  accuracy  rate  was  due  largely  to  misreading  and 
misinterpretations  of  the  text  and/or  the  questions.  EH,  the  third  nonnative  English  speaker, 
relied  heavily  on  the  text,  as  did  GL;  however  her  lack  of  science  background  prevented  her  from 
successfully  solving  type  5  questions  and  she  had  difficulty  with  type  3  questions.  Similar  to  GL, 


type  2  questions  that  required  paraphrase  and  vocabularty  conversion  presented  particular 
difficulties  for  her. 

Finally,  DW,  also  a  "low  knowledge"  learner,  showed  the  most  extensive  use  of 
monitoring  of  all  seven  learners.  She  employed  both  memory  and  text  searches  and  frequently 
employed  question  analysis.  DW  tended  to  engage  in  lengthy  text  searches  that  were  punctuated  by 
overt  consideration  of  the  question  and  the  relevance  of  specific  text  sections  .  Of  DW's  incorrect 
answers,  50%  were  partially  correct  and  reflected  inadequate  background  knowledge  with  which 
to  evaluate  her  candidate  answers. 

The  observed  differences  among  the  protocols  can  be  summarized  in  the  following  way: 
The  "expert"  native  English  learners  tended  to  rely  on  memory  searches;  the  more  successful 
learner  engaged  in  optional  answer  confirmation.  Of  the  lower  knowledge  native  English 
speakers,  one  tended  to  rely  on  memory  (LH)  and  had  a  high  rate  of  incorrect  answers.  When  she 
did  search  the  text  she  found  the  correct  answers  only  for  Type  1  (verbatim)  questions.  Finally, 

DW  relied  on  the  text  to  a  greater  degree  than  the  other  native  English  speakers.  Her  solution 
strategies  were  extensive  and  quite  often  involved  question  analysis,  monitoring  and  reasoning. 

In  general,  the  nonnative  English  speakers  relied  on  the  text  to  a  greater  degree  than  the  native 
English  speakers.  The  highest  knowledge  and  most  successful  nonnative  English  speaker,  GL, 
relied  almost  exclusively  on  the  text  and  engaged  in  frequent  question  analysis  and  monitoring 
activities.  EH  was  less  successful  in  her  efforts  to  use  the  text  and  was  hampered  frequently  by 
vocabulary/conceptual  gaps.  Finally,  II  used  memory  and  text  searches  but  was  frequently 
hampered  by  faulty  or  inadequate  question  encoding  or  text  interpretation. 

The  foregoing  characterizations  were  based  on  the  simple  presence  or  absence  of  the 
various  processing  events.  A  complimentary  analysis  considered  the  frequency  of  various 
processing  events.  The  frequency  data  index  the  length  of  solution.  Two  measures  were  computed 
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from  the  coded  protocols:  number  of  processing  events  (read,  reread,  skim,  etc.)  and  number  of 
metacognitive  processing  events  (monitor,  evaluate,  etc.).  These  data  were  used  in  conjunction 
with  the  data  in  Table  1 1  to  develop  summary  profiles  of  the  seven  learners.  These  summaries 
are  presented  in  Table  12. 

The  profiles  help  to  illustrate  the  complex  interactions  of  text,  task  and  learner  that 
were  observed.  The  two  most  successful  learners  (MR  and  GL)  differed  in  the  length  of  solution, 
with  GL  engaging  in  twice  as  many  metacognitive  and  cognitive  processing  events  as  MR.  Whereas 
GL  was  high  knowledge,  he  did  not  have  MR's  level  of  expertise  and  may  therefore  have  relied 
more  on  the  presented  information  than  did  MR.  The  two  moderately  successful  learners,  HS,  the 
content  expert,  and  DW,  a  novice  geology  student,  replicated  the  differences  between  GL  and  MR. 
Thus,  lacking  expertise  in  the  domain,  learners  who  did  well  tended  to  extensively  process  the 
written  material  in  the  context  of  the  question  and  to  keep  careful  track  of  their  performance. 

In  contrast,  the  three  less-successful  learners  had  in  common  a  tendency  to  answer 
quickly,  i.e.  to  engage  in  relatively  short  (few  step)  solutions.  Comparison  of  EH  and  II  revealed 
identical  solution  success  rates  but  that  metacognitve  processing  was  twice  as  likely  for  the  lower 
knowledge  learner.  Accuracy  was  lowest  for  the  low  knowledge,  quick  solver  who  engaged  in 
virtually  no  metacognitive  behavior. 

The  characterizations  portrayed  in  these  summary  profiles  suggest  that  successful 
learners  may  be  sensitive  to  the  need  for  strategies  that  compensate  for  low  knowledge  in  a 
domain.  Among  these  learners,  the  primary  means  of  compensation  was  relatively  lengthy  text 
processing  accompanied  by  monitoring  and  evaluation  of  the  processes  and  products  of  solution. 
Nonnative  English  speakers  who  are  sensitive  to  this  compensatory  mechanism,  and  attempt  to 
use  it.  are  heavily  dependent  on  text  negotiating  strategies  and  on  the  necessary  English  language 
skills.  In  contrast,  nonnative  English  speakers  with  high  or  expert  levels  of  knowledge  in  an  area 
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would  be  less  dependent  on  such  English  language  skills. 

Summary  and  Conclusions 

The  question-answering  model  plus  the  question  taxonomy  permit  the  analysis  of 
learners  solution  strategies  for  dealing  with  academic  text.  The  protocol  analyses  revealed  a 
complex  interaction  of  learner,  text  and  task  characteristics.  Several  important  insights  and 
tentative  conclusions  can  be  offerred.  First,  we  found  a  great  deal  of  similarity  among  the 
solution  strategies  of  the  native  and  nonnative  English  speakers,  consistent  with  previous 
examinations  of  comprehension  strategies  of  native  and  second  language  readers  (Block, 
1986a,b).  However,  one  important  difference  that  did  emerge  was  a  tendency  for  the  nonnative 
English  speakers  to  engage  in  more  text  searches  and  somewhat  more  question  analysis  than  the 
native  English  speakers,  as  a  group.  Given  the  select  and  relatively  high  proficiency  levels  of  the 
nonnative  English  speakers  sampled  in  this  study,  were  we  to  study  a  wider  spectrum  of 
nonnative  English  speakers,  we  would  anticipate  discovering  solution  strategy  differences  as  well 
as  replicating  the  difference  we  observed  here.  In  this  context,  it  is  important  to  point  out  that 
despite  extensive  efforts  to  recruit  volunteers  from  the  oceanography  course,  a  very  small 
proportion  of  the  enrollment  was  nonnative  English  speakers.  Of  these,  there  were  only  two 
other  students  who  evidenced  any  desire  to  participate  in  the  study  but  they  were  prevented  from 
doing  so  by  time  constraints. 

Solution  strategy  differences  were  related  to  expertise  in  oceanography  and  related 
science  domains.  Thus,  this  learner  characteristic  emerged  as  an  important  factor  in  question 
answering  strategies.  MR,  our  expert,  was  highly  accurate  and  relied  heavily  on  prior  knowledge 
and  memory  search  strategies.  This  type  of  solution  strategy  is  consistent  with  previous  accounts 
of  the  problem  solving  behavior  of  experts  (e.g.,  Simon  &  Simon,  1 976;  Larkin, 1 980). 

Generally,  the  more  successful  learners,  i.e.,  those  who  tended  to  get  the  correct  answers, 
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engaged  in  cognitive  monitoring,  especially  of  the  products  of  their  efforts,  i.e.,  their  answers  or 
candidate  answers.  Unsuccessful  solutions  were  associated  with  largely  unmonitored  memory 
searches  and  this  was  a  particular  obstacle  for  one  of  the  "novice"  native  speakers.  Finally,  the 
task  requirements,  captured  in  the  analysis  of  the  types  of  questions  occurring  in  the  sample, 
determined  solution  strategy  for  all  learners.  The  simplest  strategies  occurred  for  the  verbatim 
questions,  which  had  the  highest  rate  of  successful  solution.  There  were  few  differences  between 
language  groups  and/or  expert-novice  differences  for  these  questions.  The  hardest  question  type 
was  the  one  requiring  vocabulary  conversion  and/or  paraphrase  matches  between  text  and 
question.  The  expert  was  the  only  one  maintaining  highly  accurate  performance  on  these 
questions.  The  nonnative  English  speakers  ,  especially  the  low  knowledge  one  (EH),  found  these 
particularly  difficult.  The  least  successful,  low  knowledge  native  English  speaker  answered  these 
questions  incompletely  or  incorrectly  as  well. 

Our  speculation  at  this  point  regarding  the  relationship  between  knowledge,  second 
language  and  answering  questions  from  academic  text  is  that  difficulties  encountered  with  specific 
text  language  may  be  overcome  by  a  rich  knowledge  base.  A  hidden  aspect  of  expertise  in  an  area 
such  as  oceanography  is  the  familiarity  with  general  science  terms  and  stylistic  conventions 
employed  by  texts  conveying  scientific  information.  Certainly  one  area  of  particular  interest  is 
the  role  of  different  types  of  questions  in  facilitating  academic  learning.  However,  our  work 
indicates  that  questions  must  be  keyed  to  specific  relations  with  the  text  if  the  appropriate 
envisionment  level  is  to  be  attained.  Questions  that  look  like  "think"  questions  can  sometimes  be 
answered  correctly  solely  on  the  basis  of  the  text.  Other  occasions  arise  where  there  is  a  hidden 
knowledge  base  requirement  and  one  that  is  apparent  only  to  a  novice  in  the  field.  It  is  also  clear 
from  the  way  these  learners  used  the  text  that  the  method  of  choice  when  the  text  is  used  is  to  read 
and  reread  sections  that  have  a  high  number  of  words  that  match  the  wording  of  the  question. 


Further  investigations  of  the  role  of  text  signals  in  facilitating  comprehension  and  reasoning 
with  academic  material  need  to  be  pursued.  Finally,  the  degree  to  which  the  strategies  highlighted 
here  generalize  to  academic  learning  and  question  answering  with  texts  other  than  oceanography 
is  an  unknown  at  this  time. 
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Footnotes 


1 .  We  note  that  the  subjects  for  this  study  volunteered  to  participate.  The  three  nonnative 
English  speakers  were  the  only  volunteers.  Only  two  other  students  identified  themselves  to  the 
research  assistant  as  nonnative  English  speakers  but  they  declined  to  participate  due  to  the 
duration  of  the  study.  The  native  English  speakers  were  the  first  three  who  volunteered  and 
agreed  to  come  to  the  laboratory  in  a  timely  fashion  for  all  four  experimental  sessions. 

2.  From  the  original  6  questions  per  chapter,  18  separate  questions  were  scored.  One  of 
the  chapter  2  questions,  #4,  had  to  be  dropped  because  the  course  instructor  indicated  that  it  was 
a  poor  question  and  not  really  answerable  based  on  the  information  provided  in  the  text. 

3.  The  only  difference  between  solutions  for  questions  from  chapter  2  (old)  versus 
chapter  12  (new)  was  the  somewhat  higher  incidence  of  memory  and  text  searches  (“C“ 
strategies)  for  questions  on  “old"  material.  The  old-new  variable  seemed  to  not  otherwise  affect 
solution  strategies.  This  difference  is  consistent  with  the  plausible  prediction  that  questions  on 
“old"  material  would  seem  more  familiar  and  lead  to  a  greater  incidence  of  direct  retrieval  (cf. 
Reder,  1987).  However,  having  retrieved  an  answer,  there  appeared  to  be  sufficient  uncertainty 
that  subjects  treated  it  as  a  candidate  answer  and  searched  the  text  prior  to  giving  their  final 


answer. 


Subject  characteristics 


Self-rating 


learning  from  English 

command  of 

Learner 

Major 

SAT  (V/M) 

Native  Language 

lecture 

text 

science  vocab 

HSa 

Geology 

NA 

English 

Extr.  good 

Extr.  good 

Extr.  good 

MRb 

Physics 

560/650 

English 

Good 

Extr.  good 

Good 

LH 

Communications 

990-both 

English 

Extr.  good 

Extr.  good 

Gccd 

DW° 

Sociology 

550/590 

English 

Extr.  good 

Extr.  good 

Good 

II 

BusVecon 

380/540 

Croatian 

Good 

Good 

Moderate 

EH 

BusVecon. 

550/500 

Dutch 

Good 

Extr.  good 

Moderate 

GL 

History 

520/440 

Spanish 

Extr.  good 

Good 

Moderate 

aThe  teaching  assistant  for  the  course  and  a  graduate  student  studying  Oceanography. 
bMR’s  background  is  heavily  oriented  to  the  hard  sciences.  He  has  already  taken  two  other  College 
level  geology  courses.  In  combination  with  his  physics  major,  this  background  qualifies  MR  as  a 
subject  matter  expert  or  near  expert  in  the  introductory  course  in  Oceanography  because  many  of 
the  concepts  and  relationships  are  familiar  to  him  from  his  other  science  courses. 

CDW  had  had  one  year  of  foreign  study  in  Chile  and  lists  Spanish,  French  and  Russian  as  other 
languages  that  she  has  knowledge  of . 


Table  2 


Question  Answering  Model  for  Learning  from  Academic  Texts 

Processing  Events  Goals 

Encode  Question  Goal:  Determine  the  type  of  question 

Goal:  Determine  starting  point(s)  for 
searching  for  an  answer. 

Search  Memory  for  Answer  Goal:  Find  a  candidate  answer  that 

exceeds  criterion  for  response. 
(Evaluate  likelihood  of  success 
with  continued  search.  If  high, 
continue  memory  search;  If 
low,  try  another  means.) 

Goal:  Determine  alternate  sources  for 
answer.  (Memory  search  may 
provide  information  that 
facilitates  external  search.) 

Goal:  Delimit  search  space.  (Use 
question  and  results  of  any 
memory  search.) 

Goal:  Find  relevant  information. 

Goal:  Process  the  text  information 
in  the  context  of  the  task 
defined  by  the  question. 

Goal:  Answer  question  completely. 
Goal:  Match  type  of  answer  to  type  of 
question. 

Confirm  Answer 
Monitor  quality  of  answer 
Self-Monitor 


Construct  and  Output 
Answer 

Optional  Monitoring,  e.g., 


Search  External  Source  for 
Answer,  e.g.  Textbook 


Table  3 


Expansion  of  Goals  of  Question  Encoding 


Goal:  Determine  the  type  of  question. 

Does  the  question  require  explanation? 

comparison? 
definition? 
quantitative  values? 

Does  the  question  require  application  of  material  to  a  new 
situation  or  to  everyday  life,  i.e.,  life  outside  the  textbook? 

Means:  Use  the  language  in  the  question 

e.g,  How  many . ? 

What  does _ mean? 

Why....? 

Use  the  task  context  and  constraints,  e.g. 

Type  of  Situation  (homework,  in-class  exam) 
Form  (fill-in,  multiple  choice,  short  essay,  open 
ended  and  long  response  space) 

Goal:  Determine  starting  point(s)  for  search  space. 

Use  technical  terms  and  keywords  in  the  question  as  entry 
points. 


Table  4 


Interdependent  Goals  in  Textbook  Processing 


Goal:  Delimit  the  search  space. 

Means:  Use  the  results  of  question  encoding  and/or  memory 

search. 

Goal:  Find  information  relevant  to  the  question. 


Means: 


Global  Search:  Scan/skim  text.  This  may  be  exhaustive  or 
self-terminating. 

Guided  Search:  Use  question  keywords,  topic,  headings, 
subheadings  to  localize  search. 

Use  "remembered"  locations  to  localize 
search. 


Goal:  Process  the  text  information  in  the  context  of  the  task 
defined  by  the  question. 


Means:  Read,  Reread 

Rephrase,  Paraphrase 
Summarize 

Reason/Apply/Integrate 


the  text  and  the  question. 


Table  5 


DW's  protocol  for  12,6  Explanation  question 


Processing  Event 

Information 

Comment/Interpretation 

Reads 

question 

Identifies 

topic  and  text 
section 

"Latent  heat" 

Rereads 

question 

Recalls 

answer  from  memory 

Global  answer.  "Because 
of  the  evaporation  pro¬ 
cess" 

Questions 

self  -  Why  does 
process  work  that 
way? 

Describes 

strategy  =  go  to  book 

Reads 

text  B.2  S29-S32 

Reads 

text  B.2S15 

States 

answer 

Qualifies 

answer  (partially 

"1  don’t  know-  there's 

correct  answer) 

some  thing  in  the  book 
that  I'm  missing." 

Skims 

text  from  beginning 
of  chapter  through 

A.2  section 

Reads 

text  section  B.1S1- 
S3 

Skims 

text  section  B.1 

Reads 

text  section  BASI¬ 
SS 

Recognizes 

relevant  informa¬ 
tion 

on  evaporation  process 

Rereads 

text  B.2S3-S6 

Reads 

text  B.2S7-S8 

States 

answer  that 

Not  terribly  different  from 

attempts  to  para¬ 

initial  guess. 

phrase  what  she's  read 

Rereads 

text  B.2S7 

Decides/States 

answer  =  original 

Reverts  to  original  answer 

Table  6 


Interrelated  Goals  of  Answering 

Goal:  Answer  the  question  completely. 

Is  there  more  information  relevant  to  the  question? 

Has  the  extent  of  the  search  been  sufficient? 

Does  the  answer  require  further  elaboration? 

Goal:  Match  the  type  of  answer  to  the  type  of  question. 

If  the  question  asks  for  explanation 

for  comparison 

for  definition 

for  quantitative  value(s) 

has  it  been  provided? 

Goal:  Demonstrate  incorporation  into  the  knowledge  base. 

Means:  Construct  an  answer  that  does  not  match  the  textbook. 

Use  novel  examples  that  demonstrate  the  auestioned 
phenomena,  concepts,  or  principles. 


Table  7 


Types  of  Solution  Strategies 


Encode  question 


Search  types  Additional  processing  events 

♦question  analysis  +reasoning,  +  process 

inference  monitoring 

A.  memory  search, 
answer  retrieval 

A. 000  A.1000  A.0100  A.0010 

B.  text  search, 
answer  retrieval 

B. 000  B.1000  B.0100  B.0010 

C.  memory  search, 
text  search, 
answer  retrieval 

C. 0000  C.100G  C.0100  C.0010 

D.  text  search  (infor¬ 
mation  is  not  "in"  text), 
reason  beyond  text  or 
compute  answer 

D. 0000  D.1000  D.0100  D.0010 


+  product 
monitoru 

A.  0001 

B. 0001 

C. 0001 

D. 0001 


Output  answer 


Table  8 

Question  types  and  distribution  of  experimental  questions  across  types 

1.  Verbatim  relationship  between  the  question  and  the  text. 

2,3A:  "What  are..." 

12, 1A:  "How  many..."  quantitative  response 

12, 3A:  "What  fraction..."  quantitative  response 

12, 3C&D:  "What  (%)..."  quantitative  response 

12, 3E:  "Does  the  difference  (between  quantities)..." 

2.  Paraphrase  relationship  between  the  question  and  the  text, 
including  the  necessity  for  vocabulary  conversion  and 
equivalence. 


2, 1A:  "How  do  we..." 
2,  IB:  "Why  ..." 

2,5:  "How  can..." 

12,  2A:  "What  is..." 


quantitative  response 


3.  Verbatim  "look  up"  plus  comparison. 

12,  IB:  "How  does  this  compare..."  quantitative  comparison 
12, 1C:  "How  does  this  compare..."  quantitative  comparison 
12,  2B:  "How  does  this  compare..."  quantitative  comparison 
12,5:  "How  are. ..similar?"  concept  comparison 


4.  Cross  paragraph  integration. 

2,  3B:  "Explain  (properties)...'  concepts 
2,  6:  "Explain  how..." 

12, 4:  "What  is  the  difference  between..." 

12,6:  "Explain  why..." 


process 

concept  comparison 
concept  and  process. 


Reasoning,  application,  computation  questions. 

2, 2:  "How  much..."  quantitative  response.  Formula;  not  all  the 

necessary  information  is  explicitly  in  the  text. 

12, 3B:  "What..."  quantitative  response.  The  value  in  a 

series  must  be  determined.  Alternatively, 
a  rule  tha  is  given  verbatimin  the  text  (see  12.3E) 
may  be  applied. 


Strategic  Solution  Types 


K* 


v.v.v.v’.'- 


Table  10 

Strategies  for  correct  solutions  to  the  five  types  of  questions3 

Question  type 


Type  1 

Type  4 

Type  3 

Type  5 

Type  2 

Processing 

(n=35) 

(n-28) 

(n»21 ) 

(n=14) 

(n-28) 

Event 

Percent  correct 

(89%) 

(71%) 

(62%) 

(57%) 

(39%) 

Search  memory  (Type  A) 

.32 

.45 

.54 

.50 

.25 

Search  text  (Type  B) 

.58 

.40 

.31 

.25 

.53 

Search  memory 

and  text  (T ype  C) 

.10 

.15 

.15 

0 

.08 

Text  search,  reason 

beyond  (Type  D) 

0 

0 

0 

.25 

0 

No  added  processes 

.45 

.40 

.23 

.13 

.42 

Question  analysis 

.26 

.30 

.23 

.25 

.08 

Reasoning/Inference 

.13 

.05 

.54 

.63 

0 

Monitorina*3 

J2 

*S5 

M 

1.13 

Process 

.13 

.25 

.08 

.50 

.33 

Product 

.29 

.40 

.38 

.63 

.42 

aPercent  correct  solutions  is  given  in  parentheses  for  each  type  of  question.  Probabilities  in  the  body 
of  the  table  are  based  on  the  frequency  of  occurrence  of  each  event  in  correct  solutions. 

^Underlined  data  in  this  row  are  the  sums  of  the  probability  of  process  and  product  monitoring. 

Because  both  could  occur  in  a  protocol,  the  sums  can  exceed  1 .00. 


Table  1 1 

Solution  strategies  for  the  seven  learners3 


Learners 


Native  English 

Nonnative  English 

MR 

HS 

CW 

LH 

gl 

BH 

II 

Correct 

16 

12 

12 

8 

15 

10 

1 

(89%) 

(67%) 

(67%) 

(44%) 

(83%) 

(55%) 

(55' 

Search  Type 

Memory 

9(8) 

10(7) 

5(4) 

10(5) 

3(2) 

3(3) 

7  (£ 

Text 

5(4) 

7(5) 

7(4) 

7(3) 

15  (13) 

10(5) 

9(5 

Both 

3(3) 

— 

4(2) 

1  (0) 

... 

4(2) 

1  (1 

Compute 

1  (1) 

1  (0) 

1  (1) 

— 

... 

1  (0) 

1  (C 

Added  Processing  Events 

None 

7 

5 

2 

7 

1 

5 

Question  analysis  2 

2 

9 

8 

9 

6 

Reasoning/inference  5 

6 

6 

5 

6 

4 

Monitoring13 

2 

12 

H 

Z 

11 

2 

Process 

3 

8 

9 

3 

6 

8 

Product 

8 

3 

9 

4 

6 

3 

aMaximum  -  18.  Frequencies  in  parentheses  are  the  frequencies  of  correct  solutions. 
bThese  data  are  the  number  of  solutions  containing  either  process  or  product  monitoring. 


Table  12 


Summary  profiles  of  the  seven  oceanography  learners 


Accuracy 

Knowledge 

Speed  of 
solution 

Successful  learners 

M=t 

High 

Expert 

Quick 

(89%) 

(67  events) 

GLa 

High 

High 

Slow 

(83%) 

(148  events) 

Moderately  successful  learners 

HS 

Moderate 

Expert 

Medium 

(67%) 

(92  events) 

DM 

Moderate 

Low 

Slow 

(67%) 

(172  events) 

Less  successful  learners 

lla 

Low 

Medium 

Quick 

(55%) 

(90  events) 

EHa 

Low 

Low 

Quick 

(55%) 

(83  events) 

LH 

Low 

Low 

Quick 

(44%) 

(80  events) 

aNonnatrve  English  speakers 

Monitoring 

Moderate 
(27  events) 
High 

(46  events) 

Medium 
(31  events) 
High 

(76  events) 

Low 

(17  events) 
Medium 
(32  events) 
Low 

(9  events) 
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Chapter  2, 1 : 

How  do  we  try  to  simulate  conditions  deep  within  the 
earth?  b)  Why  is  this  not  very  satisfactory? 

Answer  according  to  instructor:  a)  Using  a  pressure  vessel  with 
an  explosive,  b)  This  is  not  very  satisfactory  because  with  the 
explosive,  the  high  temperatures  and  pressures  are  created  only 
for  an  instant.  It  is  not  a  stable  equilibrium  situation  as  exists 
within  the  earth,  and  whatever  measurements  are  to  be  made 
must  be  done  instantaneously. 

The  answer  is  given  in  the  very  first  paragraph  of  the  section  they  were 
given  to  read:  Part  A  is  answered  in  BS5;  part  B  in  BS6,  7.  However,  the 
term  simulate  is  not  explicitly  used.  Thus,  the  student  needs  to 
understand  the  concept  simulate  and  to  recognize  that  a  pressure  vessel 
and  an  explosive,  referred  to  in  S5  would  constitute  such  a  simulation. 
Simulation  must  be  distinguished  from  the  indirect  and  the  direct  methods 
of  study  that  are  discussed  in  the  same  section  of  the  text.  The  answer  to 
part  B  is  relatively  easier  to  find  because  the  text  language  conveys  the 
dissatisfaction,  e.g.  Even  if  ...in  comparison  to  ...  within  the  earth; 
...different  propertion  than  ...within  the  earth. 

A  confusion  on  the  part  of  a  number  of  the  students  appeared  when 
they  tried  to  answer  this  question.  A  number  of  the  answers  indicated 
that  indirect  methods  were  inadequate  for  the  reasons  given  for  the 
simulation  being  inadequate.  This  interpretation  actually  contradicts  the 
text.  In  fact,  indirect  methods  are  the  main  ways  in  which  the  interior  of 
the  earth  is  studied.  The  rest  of  the  chapter  goes  on  to  discuss  these 
indirect  methods  (Gravity  studies,  seismic  waves,  etc.). 

Chapter  2,  2: 

The  moon  has  1/80  as  much  mass  as  the  earth  and  1/4  the 
radius.  How  much  would  you  weigh  on  the  moon? 

The  text  gives  a  formula  that  is  needed  to  answer  this  question  (Sect.  B.l 
Si -4).  The  definitions  given  for  the  terms  in  the  equation  do  not  match 
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the  question  on  the  face  of  it  so  the  substitutions  that  are  needed  are  not 
transparent.  For  example,  r  is  defined  as  the  separation  between  two 
objects  as  well  as  the  distance  between  to  objects.  How  this  relates  to 
the  question  is  not  clear  unless  you  know  that  r  may  be  set  equal  to  the 
earth's  radius.  Also,  it  is  necessary  to  realize  that  the  Gs  can  be  ignored 
since  they  are  constants. 

Chapter  2,3: 

What  are  two  properties  of  seismic  waves  that  tell  us 
what  the  earth's  interior  is  like?  b)  Explain  them. 

The  two  properties  are  reflection  and  refraction.  They  are 
explicitly  marked  as  properties  in  the  text:  B.2S4,  B.2S5  and  B.2S12.  They 
text  provides  explanations  for  each.  For  reflection,  the  text  gives  a 
comparison  to  lightwaves  bending  B.2S6;  B.2S7,  8,  9  and  10  elaborate  this 
example  and  discuss  how  the  reflection  property  permits  one  to  determine 
the  internal  structure  of  the  earth.  For  refraction,  the  text  gives  an 
analogy  to  a  car  slipping  off  a  highway  (B.2S13);  the  next  two  paragraphs 
discuss  the  details  of  how  refraction  works  to  reveal  structure  (B.2S1 4  - 
S24).  There  are  two  figures  illustrating  reflection  (Fig.  2.9)  and 
refraction  (Fig.  2.10  -  Snell's  law.). 


[This  section  is  followed  by  the  introduction  of  two  types  of  waves 
(S  and  P  waves).  The  two  types  are  differentially  sensitive  to  materials 
in  the  earth's  interior.  Many  students  gave  two  types  of  waves  instead  of 
two  properties.  No  students  really  got  into  the  details  of  the  refraction 
property. 


Chapter  2,5: 

How  can  satellites  be  used  to  get  information  regarding 
subsurface  mass  distributions? 


;• 

Answer  according  to  instructor:  Any  deviation  from  a 
*!  spherically-symmetric  distribution  of  masses  within  the  earth 

2  will  cause  detectable  changes  in  the  satellite's  orbit  and  speed. 
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This  answer  is  virtually  verbatim  from  the  text:  B.1  SI  2.  However,  the 
meaning  of  this  sentence  is  not  transparent  and  the  text  does  not  use  the 
term  "subsurface  mass  distribution"  in  the  body  of  the  text.  There  is  a 
Figure  (2.8)  right  above  this  sentence  that  uses  the  term  "subsurface  mass 
distibution"  and  connects  smd  to  gravitational  fields  at  the  surface.  The 
student  must  infer  the  relationship  between  orbiting  satellites  and 
surface  gravity  and  how  variations  in  gravity  affect  the  shape  of  the  orbit 
and  the  speed  of  the  satellite. 

The  text  mentions  one  device,  a  gravitometer  (B.1  S5),  that  is  used 
to  measure  changes  in  the  earth's  gravity  from  the  average  value.  Also  in 
B.1  SI  3  the  text  refers  to  altimeters,  which  are  instruments  that  map 
suface  topography,  not  subsurface  masses.  "Using  altimeters"  or  "with 
altimeters"  is  an  incorrect  answer  by  itself.  In  combination  with  a 
discussion  of  gravity  and  affects  of  subsurface  mass  on  surface  gravity,  it 
was  taken  as  correct. 

Chapter  2,  6: 

Briefly  explain  how  "echoes”  can  be  used  to  measure  the 
depths  of  discontinuities  in  the  earth’s  internal  structure. 

Answer  according  the  the  instructor:  Seismic  disturbances  create 
waves,  some  of  which  reflect  off  of  subsurface  interfaces  and  come  back 
toward  the  surface.  As  these  "echoes"  arrive  at  the  surface,  later  arrival 
times  mean  they  have  traveled  further,  indicating  deeper  interfaces. 
Knowing  time  and  speed,  we  can  calculate  distance. 

-There  is  a  simple  and  a  complex  answer  to  this  question.  The 
simple  deals  only  with  the  time  for  the  echo  to  come  back.  The  complex 
answer  deals  with  time  and  speed. 

Relevant  portions  of  the  text:  B.2  S4  -  SI  0  plus  Figure  2.9;  B.2  S9 
gives  information  for  the  simple  answer.  B.2  SI  9  -  25  deals  with  time  and 
speed  relationship. 

Need  to  recognize  that  discontinuities  is  the  same  as  interfaces 
between  materials. 


Chapter  12,  Question  #1 : 


If  all  the  water  in  the  atmosphere  were  to  come  out  in  one 
worldwide  rainstorm,  how  many  inches  (or  centimeters)  of 
rainfall  would  it  amount  to?  How  does  this  compare  to  the 
water  in  the  oceans?  the  ground  water? 

Answer  supplied  by  instructor: 

3cm.  This  is  about  1/100,000  as  much  water  as  are  in  the 
oceans  and  about  1/360  as  much  water  as  is  in  ground  water. 

Sentence  5  in  Section  A.1  contains  3cm  answer.  Verbatim  from  text.  Need 
to  use  Table  12.1  (pg.  324)  to  answer  second  part  of  the  question. 

Chapter  12,  Question  #2: 

(a)  What  is  the  latent  heat  of  vaporization  of  1  g  of  water? 

(b)  How  does  this  compare  to  the  heat  required  to  bring  it  from 
freezing  to  boiling? 

Answer  given  by  instructor: 

(a)  540  calories 

(b)  5.4  times  as  great  (100  calories  are  required  to  bring 
it  from  freezing  to  boiling  and  540  calories  are  needed  to 
vaporize  it  at  100°C). 

Answer  is  given  in  SI  -  S4  under  B.2  heading,  (page  325,  first  paragraph 
under  heading.) 

Must  understand 

1 )  that  evaporate  =  vaporization 

2)  that  latent  heat  of  vaporization  =  the  amount  of  calories  needed 
to  evaporate  water  after  it  reaches  boiling.  This  is  not  explicitly  stated 

in  the  text.  Either  have  to  infer  it  or  know  it  beforehand  (prior 
knowledge). 

Critical  Text  Language:  additional  540  calories  to  evaporate. 

Definition  given  in  text  (S28  under  3.2):  Latent  heat  =  energy  in  an 
evaporated  water  molecule,  or  energy  available  to  be  released  when 
evaporated  water  molecule  condenses. 


t  / 


To  answer  part  b,  must  make  a  comparison  between  540  and  100. 


Chapter  12,  #3  Question: 

a)  Suppose  air  at  25°  C  is  saturated  (100%  relative 
humidity)  what  fraction  of  the  air  is  water? 

b)  What  would  be  the  answer  if  the  temperature  were  35°? 

c)  15°? 

d)  5°? 

e)  Does  the  answer  change  by  roughly  a  factor  of  2  for 
every  10°? 

Answers  given  by  instructor: 

a)  3.1%;  b)About  6%;  c)1.7 %(1.68%)\  d)  .9%  (.86%) 
e)  Yes  (actually  just  slightly  less  than  a  factor  of  2). 

Answer  to 

a)  is  given  in  a  text  statement  (Under  B.  1,  Sentence  7 p.  325)  and 
Table  12.2,  p.325. 

b)  must  be  computed  either  by  interpolation  using  Table  12.2  or  by 
remembering  the  answer  to  e)  and  applying  it  to  get  the  answer  for  35° 
from  25°. 

e)  given  verbatim  in  text  -  Sentence  3  under  the  B.1  heading. 

c)  and  d)  are  read  directly  from  Table  12.2. 

Answering  this  question  also  involves  knowing  what  is  not  relevant.  The 
text  following  Sentence  7  has  many  irrelevant  numbers  in  it,  as  well  as 
discussing  at  length  issues  associated  with  relative  humidity.  Also, 
understanding  the  vocabulary  term  factor  is  necessary. 

Chapter  12,  #4  Question: 

What  is  the  difference  between  "sensible  heat"  aand 
"latent  heat"  in  the  air? 


Answer  given  by  instructor: 


Sensible  heat  is  the  heat  that  goes  into  raising  the 
temperature  of  the  air  without  increasing  the  water  content, 
and  latent  heat  is  the  heat  that  goes  into  evaporating  more 
water  into  the  air. 

The  answer  is  given  directly  in  the  text  One  paragraph  (Section  B.2, 
paragraph  3)  mentions  the  two  concepts  first  and  puts  them  in  "  "s.  It 
defines  them  differently  from  the  answer  given  by  the  instructor-or  so  it 
would  seem.  Then  the  next  paragraph  that  starts  with  a  "Suppose"  and 
might  suggest  that  a  new  topic  is  underway  actually  continues  with  an 
illustration  and  elaboration  of  the  definitions  and  difference.  The  second 
sentence  of  that  paragraph  gives  the  answer  that  matches  the  instructor's 
(S30).  Then  in  S31  an  additional  explicit  difference  is  given  ("Although 
the  required  amount  of  sensible  heat  does  not  depend  on  the  initial 
temperature  of  the  air,  the  amount  of  latent  heat  does.") 

Chapter  12,  Question  #5  : 

How  are  the  oceans  and  the  atmosphere  similar  to  sheets 
of  paper? 

Answer:  Ocean  and  atmosphere  are  only  a  few  kilometers  thick 
but  go  completely  around  the  earth.  Their  relative  thickness  in 
comparison  to  their  breadth  is  comparable  to  that  of  sheets  of 
paper.  They  are  also  as  interconnected  as  two  successive  pages 
in  a  closed  book. 

Answer  comes  from  the  second  paragraph  at  the  beginning  of  the  chapter 
(page  323).  This  is  a  verbatim  question. 

Chapter  12,  Question  #6: 

Explain  why  you  feel  cold  when  you  get  out  of  the  shower. 

Answer:  Faster  moving  molecules  leave  skin  and  enter 
atmosphere,  taking  their  energy  and  heat  with  them  (they 
evaporate);  slower  moving  are  left  behind.  The  slower  moving 
have  less  energy  and  heat  than  the  ones  that  leave. 
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Answer  is  given  in  Section  B.2  Latent  heat  of  vaporization  but  relevant 
information  appears  to  be  given  in  several  different  sections.  At  the  end 
of  the  first  paragraph  (S7,  8);  in  the  second  paragraph  and  then  again  in  the 
third.  Each  time,  the  language  is  a  little  different.  Actually  what  the  text 
does  is  to  first  state  why  we  perspire  (S7  &  8).  Then  in  the  next  paragraph 
(S9  - 14),  the  scientific  details  of  the  process  are  given.  Then  in  the  third 
paragraph,  the  text  makes  the  connection  between  the  scientifically 
described  process  and  the  everyday  experience  (SI  5  -  Si  8). 

(S  9  - 14  explains  process  underlying  molecule  movement.) 
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Appendix  B 

Codes  used  in  analyzing  question  answering  protocols 

The  protocols  were  coded  in  terms  of  the  cognitive  (or 
metacognitive)  action  that  was  being  carried  out  in  reference  to  a  type  of 
information.  Actions  appearing  on  the  same  line  (e.g.,  skim/scan)  are 
functionally  equivalent  operations. 

Action  or  Operation  Information  Type 


Reads 

Rereads 

Paraphrases 

Summarizes 

Skims/scans 

Searches 

States 

Re-states/repeats 

Elaborates/Emphasizes 

Qualifies 

Describes  (internal  state) 

Questions 

Guesses 

Recalls/Retrieves 

Recognizes/Identifies 

Locates/Finds 

Compares/Detects 

Infers/Reasons 

Rejects 

Concludes 

Monitors 

Evaluates 

Confirms 

Justifies 


Text  material 
Table,  chart  or  figure 
Question 

Long  term  memory 
Information  from  long  term 
memory 

Inferred  information/conclusion 
Own  cognitive  activity  (including 
goals,  obstacles,  strategies) 
Answer/candidate  answer 
Headings,  Subheadings,  boldface 
terms 


Of.  Phillip  L.  Ackerman 
University  of  Minnesota 
Department  of  Psychology 
Minneapolis,  MN  55455 


Dr.  Beth  Adelson 

Department  of  Computer  Science 
Tufts  University 
Medford,  MA  021 55 


AFOSR, 

Life  Sciences  Directorate 
Bolling  Air  Force  Base 
Washington,  DC  20332 


Dr.  RcOert  Ahlers 
Human  Factors  lab..  Code  N7i  i 
Naval  Training  Systems  Center 
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